Connections between Fingerspelling and Print: The Impact of Working Memory and Temporal Dynamics on Lexical Activation

Recently there has been a renewed interest in characterizing the role of fingerspelling for deaf readers. The present study takes a step back and creates a theoretical foundation for investigating similarities between fingerspelling and print decoding in hearing signers. In this way, we can probe the constraints of temporal processing and memory on L1 orthography and the processing of L2 fingerspelling. Using a cross-modal priming paradigm, the role of orthography and phonology in print and fingerspelling word recognition was investigated. Results indicate significant inhibition in target retrieval when the prime was fingerspelled but not when it was presented in print. It was hypothesized that inhibition was due to either recoding or prime temporal dynamics. Hearing nonsigners were tested with serially or simultaneously presented print to determine the role of recoding and temporal dynamics. The results suggest that: (1) difficulties in Joshua T. Williams is a Ph.D. researcher in Psychological and Brain Sciences, Cognitive Science, and Speech and Hearing Sciences at Indiana University. Sharlene D. Newman is an associate professor in Psychological and Brain Sciences at Indiana University. 158 | Sign Language Studie s processing fingerspelling for L2 learners might arise from recoding back into an L1 orthographic representation; (2) working memory abilities may reduce inhibition caused by recoding in L2 learners; (3) serial presentation of an orthographic code, either manual or visual, reduces priming effects; and (4) letter position differences provide evidence of depletion of activation over time. Deaf students have scored lower than their hearing counterparts on literacy tests for decades; their median reading level upon high school graduation is equivalent to the fourth grade (Traxler 2000). Studies have debated whether phonemic awareness (Ehri et al. 2001; Liberman and Shankweiler 1985; Snow, Burns, and Griffin 1998), phonological awareness (Paul, Wang, Trezek, and Luckner 2009; Wang, Trezek, Luckner, and Paul 2008), signing ability (Mayberry, del Guidice, and Lieberman 2011), articulatory mechanisms (Hirsh-Pasek and Treiman 1982), visual codes (McQuarrie and Parrila 2009), or orthographic competence (Miller 2004) should be targeted in order to improve deaf reading. Researchers have recently begun reinvestigating fingerspelling as an alternative code; however, only a relatively small number of these studies have investigated the role of fingerspelling in literacy. Previous studies have examined the parsing of fingerspelling (Emmorey and Petrich 2011; Patrie 1990, 1992), acquisition of fingerspelled vocabulary (Happtonstall-Nykaza and Schick 2007), and identification, matching, and spelling/writing of fingerspelled words (Puente, Alvarado, and Herrera 2006). The current study is a first step in investigating the relationship between fingerspelling and print during lexical decision making to explore how activation spreads between these representations. Here, hearing nonnative second-language (L2) learners were examined. Fingerspelling is the sequential letter-by-letter handshape construction of words in American Sign Language (ASL), with characteristics similar to those of the written alphabet (Padden and Gunsauls 2003; Patrie 1990). That is, one could think of fingerspelling as an orthographic system in ASL. Often fingerspelling is used to nominalize English labels such as places and names (Padden 1998). Fingerspelling resides in an integrated subspace of the native ASL lexicon and enhances the ability to share semantic representations with signs Connections between Fingerspelling and Print | 159 (Brentari and Eccarius 2010). Furthermore, it is frequently used in everyday ASL (Padden and Gunsauls 2003). Fingerspelling serves as a cross-modal bridge between ASL and English in that it is a manual representation of the English alphabet (see figure 1), while also having some correspondence to ASL signs. The connections to both English print and ASL signs may allow fingerspelling to aid in literacy acquisition in deaf students by connecting fingerspelled words, print words, and ASL signs (i.e., “chaining”; see Chamberlain and Mayberry 2000). Johnson (1994) delineated many of the ways in which fingerspelling connects to literacy acquisition in children. Of particular interest, he notes that deaf children not only acquire fingerspelling early (also see Maestas y Moores 1980; Padden 1990) but also learn associations between the alphabet and fingerspelling very early (also see Williams 1976). These associations have also predicted later literacy performance (Allen 2015; Mayberry, del Giudice, and Lieberman 2011). Additionally, combining fingerspelling, print, and ASL signs in a pedagogical context might not only be advantageous for native deaf signers ( Johnson 1994) but also assist L2 acquisition for hearing learners of ASL, given their inability to easily acquire and process fingerspelling (Patrie 1990; Wilcox 1988). As such, it is important to understand the connections between print and fingerspelling processing. Fingerspelling may prove to be important in literacy acquisition for deaf persons. Some evidence from the literature supports the hypothesis that fingerspelling has positive connections to orthography and reading (Mayberry, del Giudice, and Lieberman 2011; Padden 2006; Padden and Ramsey 2000; Hanson 1982). Although these studies Figure 1. The top row demonstrates a fingerspelled string, s-a-i-l , for the English word sail, with the corresponding English letters on the bottom row. The fingerspelled stills were extracted from the native signer stimuli as described in the Methods section. 160 | Sign Language Studie s provide a cache of evidence that fingerspelling may facilitate learning to read, Emmorey and Petrich (2011) investigated the association between print and fingerspelling by examining the segmentation of print words and fingerspelled words. The authors examined basic orthographic syllable structure (BOSS; Taft 1979), the parsing of print, and fingerspelling by deaf adults. The BOSS model suggests that ortho graphic decoding is optimized by a perceptual boundary after nonviolating orthotactic postvocalic consonants (e.g., cand.le for “candle,” such that “dl” is an illegal coda cluster). The authors hypothesized that if fingerspelling activates print representations, then similar segmentation strategies would be implemented. They found that deaf participants parse print and fingerspelling differently. Print was parsed using BOSS segmentation, whereas fingerspelling was parsed using a phonological strategy. The difference in segmentation in print and fingerspelling raises doubt about the activation of print representations from fingerspelling. It may also be the case that the inherent differences in presentation cause differences in segmentation. Differences in the processing and activation of print and fingerspelling may account for difficulties in fingerspelling comprehension in L2 learners. Several studies have demonstrated that the ortho graphic system of the first language (L1) influences acquisition and word recognition of L2 orthography. Moreover, Korean-English learners showed more errors in judging homophone foils in English word recognition, whereas Chinese learners made more errors on orthographically similar items than controls (Wang, Koda, and Perfetti 2003). These results suggest that Korean-English learners relied more on phonological information and Chinese-English learners relied more on orthographic information. It was hypothesized that the functional overlap between Korean and English orthographies facilitated a phono logical processing strategy, whereas the divergence between English and Chinese orthographies afforded an orthographic processing strategy. Based on this study and others (Koda 1999, 1997, 1989; Chikamatsu 1996), it can be concluded that languages have different affordances for mapping, which subsequently influence L2 acquisition and lexical processing. Some languages, such as English, rely on phoneme-grapheme mappings, while other languages select other units (e.g., syllables or morphemes). More importantly, the differences between L1 and L2 in Connections between Fingerspelling and Print | 161 their alphabetic mapping strategies can influence L2 word recognition. It is the case that ASL learners would have similar strategies in both L1 (English) and L2 (ASL) because both fingerspelling and print are alphabetic systems. Additionally, priming studies support evidence of cross-linguistic interactions between phonological and orthographic systems. Bilinguals and L2 learners can use both phonological and orthographic information during visual word recognition (Dimitropoulou, Duñabeitia, and Carreiras 2011; Brysbaert 2003). Orthographic and phonological priming studies of monolingual English speakers suggest that overlapping orthographic and/or phonological information facilitates word recognition (Ferrand and Grainger 1994; Lukatela and Turvey 1990). One major difference between English and ASL, unlike English and Chinese, for example, is the sequentiality of letter presentation in ASL fingerspelling, which might significantly influence the learners’ strategy during visual word recognition (Patrie 1990, 1992). Taken together, evidence from cross-linguistic word recognition and priming tasks suggest that it is plausible to assume that hearing learners of ASL would use both phonological and orthographic information while making lexical decisions for both English print and ASL fingerspelling; however, differences in word recognition might emerge due to the temporal dynamics of fingerspelling. Patrie and colleagues have previously investigated how the temporal dynamics of fingerspelling may affect visual word recognition of fingerspelling by novice and expert hearing signers. Using a rapid serial visual processing task (Patrie and Johnson 2011), it was found that experts were better able to recognize fingerspelled pseudowords than novices (Patrie 1990, 1992). Of particular importance to the present study is that Patrie (1990) tested the ability of novices and experts to recognize both fingerspelling and computer-spelled words (i.e., words that were presented one letter at a time on the computer). Patrie (ibid.) found that novice signers recognized more computer-spelled words than fingerspelled words, but the converse was true for experts. This pattern of findings suggested that experience with the serial presentation of letters (i.e., the temporal dynamics of finger spelling) directly affects orthographic processing. Given this interaction between fingerspelling and serial letter processing, we are still left to wonder whether print and fingerspelling share similar processing mechanisms. 162 | Sign Language Studie s This study tested the hypothesized ability of a fingerspelling representation to activate print representations and vice versa in hearing signers. In a previous study (Williams, Darcy, and Newman 2015), it was found that the fingerspelling of primes does not facilitate fingerspelled target selection in L2 learners of ASL but that deaf signers did show facilitation in the orthographically related conditions. The priming effects elicited in deaf signers was not surprising as it was within-language priming, where representations are expected to share activation. However, the result for L2 learners was a bit more surprising due to their robust L1 orthographic representations, which were expected to aid in fingerspelling activation. Hearing L2 learners should have been able to bootstrap fully specified orthographic knowledge to aid in fingerspelling retrieval. Thus, it was concluded that the serial nature of fingerspelling must differentially affect lexical retrieval in L2 learners. The current study aimed to explicate the role of temporal dynamics in fingerspelling activation in L2 learners. A cross-modal priming paradigm was used to determine whether print primes could facilitate the lexical retrieval of a fingerspelled target. If fingerspelling representations activate print representations, then significant facilitation should result when making a lexical decision with regard to print words. Similarly, it was hypothesized that print would activate fingerspelling representations and facilitate lexical decisions about fingerspelling. The study also manipulated phonology to investigate phonological parsing strategies. It was predicted that phonology would influence lexical decision making because phonology is automatically activated with orthography for L1 hearing individuals (Grainger and Ferrand 1994). Based on the previous evidence of a preferred phonological parsing strategy for fingerspelling (for deaf signers), it was hypothesized that, when the prime was fingerspelled, the phonological activation would be greatest and would affect the print target. However, the print prime would activate the phonology less significantly and would not influence the retrieval of the fingerspelled prime. It was hypothesized that the greatest priming effects would occur in conditions where the prime and the target, regardless of whether print or fingerspelling, overlap in orthography, not when they overlap in phonology. Connections between Fingerspelling and Print | 163 Experiment 1: Fingerspelling and Print Cross-Modal Priming